{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cKzm8_U2aJNR"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/facebookresearch/fairo/blob/master/tutorials/semantic_parser_onboarding.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VVCwKqEckiVq"
   },
   "source": [
    "# Semantic Parser Onboarding\n",
    "\n",
    "The **semantic parser** is a seq-to-seq model built on the Huggingface Transformers library. The input to the parser is a chat command, eg. \"build a red cube\". The output is a linearized parse tree (see [Action Dictionary Spec Doc](https://github.com/facebookresearch/fairo/blob/main/base_agent/documents/Action_Dictionary_Spec.md) for the grammar specification).\n",
    "\n",
    "The encoder uses a pretrained DistilBERT model, followed by a highway transformation. For the default model, encoder parameters are frozen during training. The decoder consists of a 6-layer Transformer, and has a **Language Modeling** head, **span** beginning and span end heads, and **text span** beginning and end heads. The Language Modeling head predicts the next node in the linearized tree. The span heads predict the span range, which provides the value for the span node. For more details, see the [Craftassist Paper](https://www.aclweb.org/anthology/2020.acl-main.427.pdf).\n",
    "\n",
    "This tutorial covers the end-to-end process of how to train a semantic parser model and use it in the CraftAssist agent:\n",
    "\n",
    "*  Generating and preparing datasets\n",
    "*  Training models\n",
    "* Evaluating models\n",
    "* Using models in the agent\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4ygCXenKCi8A"
   },
   "source": [
    "## Set Up\n",
    "\n",
    "### Downloading Pre-Trained Models and Datasets\n",
    "\n",
    "When you run the CraftAssist agent for the first time, the pre-trained models and data files required by the project are downloaded automatically from S3.\n",
    "\n",
    "```\n",
    "cd ~/minecraft/craftassist\n",
    "python ./agent/craftassist_agent.py\n",
    "```\n",
    "\n",
    "You can also do this manually:\n",
    "\n",
    "```\n",
    "cd ~/minecraft\n",
    "./tools/data_scripts/try_download.sh\n",
    "```\n",
    "\n",
    "This script checks your local paths `craftassist/agent/models` and `craftassist/agent/datasets` for updates, and downloads the files from S3 if your local files are missing or outdated (optional).\n",
    "\n",
    "### Conda Env\n",
    "\n",
    "You may need to upgrade/downgrade your pytorch and CUDA versions based on your GPU driver.\n",
    "\n",
    "For a list of pytorch and CUDA compatible versions, see: https://pytorch.org/get-started/previous-versions/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5h6Nr7Dx1SSP"
   },
   "source": [
    "## Datasets\n",
    "\n",
    "The datasets we use to train the semantic parsing models consist of:\n",
    "* **Templated**: This file has 800K dialogue, action dictionary pairs generated using our generation script.\n",
    "    * **Templated Modify**: This file has 100K dialogue, action dictionary pairs generated in the same way as templated.txt, except covering modify type commands, eg. \"make this hole larger\".\n",
    "* **Annotated**: This file contains 7k dialogue, action dictionary pairs. These are human labelled examples obtained from crowd sourced tasks and in game interactions.\n",
    "\n",
    "See the CraftAssist paper for more information on how datasets are collected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mBDOjbOXAo3d"
   },
   "source": [
    "We provide all the dialogue datasets we use in the CraftAssist project in a public S3 folder: \n",
    "https://craftassist.s3-us-west-2.amazonaws.com/pubr/dialogue_data.tar.gz\n",
    "\n",
    "In addition to the datasets used to train the model, this folder also contains greetings and short commands that the agent queries during gameplay."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U0SPXWga38M7"
   },
   "source": [
    "### Generating Datasets\n",
    "\n",
    "This section describes how to use our tools to generate and process training data.\n",
    "\n",
    "To generate some templated data to train the model on, run ``generate_dialogue.py``. This script generates language commands and their corresponding logical forms using heuristic rules and publicly available dialogue datasets. \n",
    "\n",
    "Provide the number of examples you want to generate, eg. for 500K examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9gpKV_bjkIc_",
    "outputId": "00b670f6-0c2f-4bac-cc2d-642e30a46a4f"
   },
   "outputs": [],
   "source": [
    "! cd ~/minecraft/base_agent/ttad/generation_dialogues\n",
    "! python generate_dialogue.py -n 500000 > generated_dialogues.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mirbvp1_4sUU"
   },
   "source": [
    "This creates a text file. We next pre-process the data into the format required by the training script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TkYMEoYP4yG5",
    "outputId": "dbb4965d-1220-4162-f86e-fa5cc106c986"
   },
   "outputs": [],
   "source": [
    "! cd ../ttad_transformer_model/\n",
    "! python ~/droidlet/tools/nsp_scripts/data_processing_scripts/preprocess_templated.py \\\n",
    "--raw_data_path ../generation_dialogues/generated_dialogues.txt \\\n",
    "--output_path [OUTPUT_PATH (file must be named templated.txt)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_r_PxlAaAbPA"
   },
   "source": [
    "The format of each row is \n",
    "```\n",
    "[TEXT]|[ACTION DICTIONARY]\n",
    "```\n",
    "\n",
    "To create train/test/valid splits of the data, run\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VJBMiU_o-loR",
    "outputId": "2e41ea60-05f2-4099-d249-64823810f539"
   },
   "outputs": [],
   "source": [
    "! python ~/droidlet/tools/nsp_scripts/data_processing_scripts/create_annotated_split.py \\\n",
    "--raw_data_path [PATH_TO_DATA_DIR] \\\n",
    "--output_path [PATH_TO_SPLIT_FOLDERS] \\\n",
    "--filename \"templated.txt\" \\\n",
    "--split_ratio \"0.7:0.2:0.1\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m-E-iLb8eGNe"
   },
   "source": [
    "To create a split of annotated data too, simply run the above, but with filename \"annotated.txt\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NlrcJguQ_FxW"
   },
   "source": [
    "We are now ready to train the model with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "l9DtuFQwkEGg",
    "outputId": "14dfc729-ec9e-4ca3-e1ce-6a8417d7e534"
   },
   "outputs": [],
   "source": [
    "! cd ~/minecraft\n",
    "! python base_agent/ttad/ttad_transformer_model/train_model.py \\\n",
    "--data_dir craftassist/agent/models/ttad_bert_updated/annotated_data/ \\\n",
    "--dtype_samples '[[\"templated\", 0.35], [\"templated_modify\", 0.05], [\"annotated\", 0.6]]' \\\n",
    "--tree_voc_file craftassist/agent/models/ttad_bert_updated/models/caip_test_model_tree.json \\\n",
    "--output_dir $CHECKPOINT_PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to experiment with the model parameters. Note that ``dtype_samples`` is the sampling proportions of the different data types. ``templated`` is generated using the ``generate_dialogue`` script as described above, whereas ``annotated`` is obtained from human labellers.\n",
    "\n",
    "With a single NVIDIA Quadro GP100 GPU, one training epoch typically takes 30 minutes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WCh8Dot4_NcQ"
   },
   "source": [
    "The models and tree vocabulary files are saved under ``$CHECKPOINT_PATH``, along with a log that contains training and validation accuracies after every epoch. Once you're done, you can choose which epoch you want the parameters for, and use that model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cYRQmBV3-zSO"
   },
   "source": [
    "You can take the params of the best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bx04vfC8_bZW",
    "outputId": "682ec505-1ae7-4616-94cf-1958f5de416a"
   },
   "outputs": [],
   "source": [
    "! cp $PATH_TO_BEST_CHECKPOINT_MODEL craftassist/agent/models/caip_test_model.pth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f0z1Rih4HJA6"
   },
   "source": [
    "## Testing Models\n",
    "\n",
    "During training, validation accuracy after every epoch is calculated and logged. You can access the log file in the output directory, where the checkpointed models are also saved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_EICCQmMIMXE"
   },
   "source": [
    "You can test the model using our inference script:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y2Ye2CZsH8qm"
   },
   "outputs": [],
   "source": [
    "! python3 -i ~/droidlet/tools/nsp_scripts/data_processing_scripts/test_model_script.py\n",
    ">>> get_beam_tree(\"build a house\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will output the logical form for this command, i.e."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 164
    },
    "id": "__n2-rpTll4z",
    "outputId": "3b14e779-9c8d-461f-9885-1d2fa11007a5"
   },
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "pprint({'dialogue_type': 'HUMAN_GIVE_COMMAND', 'action_sequence': [{'action_type': 'BUILD', 'schematic': {'has_name': [0, [2, 2]], 'text_span': [0, [2, 2]]}}]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate accuracy on a test dataset, eg. annotated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NZcaRqK5ljR5"
   },
   "outputs": [],
   "source": [
    ">>> model_trainer = ModelTrainer(args)\n",
    ">>> full_tree_voc = (full_tree, tree_i2w)\n",
    ">>> model_trainer.eval_model_on_dataset(encoder_decoder, \"annotated\", full_tree_voc, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XqEmaJJ-_gfm"
   },
   "source": [
    "You can now use this model to run the agents. Some command line params to note:\n",
    "\n",
    "`--dev`: Disables automatic model/dataset downloads.\n",
    "\n",
    "`--ground_truth_data_dir`: Path to folder of ground truth short commands and templated commands. When given a command, the agent first queries this set for an exact match. If it exists, the agent returns the action dictionary from ground truth. Otherwise, the agent queries the semantic parsing model. Defaults to `~/minecraft/craftassist/agent/datasets/ground_truth/`. You can write your own templated examples and add them to `~/minecraft/craftassist/agent/datasets/ground_truth/datasets/`.\n",
    "\n",
    "`--nsp_models_dir`: Path to binarized models and vocabulary files. Defaults to `~/minecraft/craftassist/agent/models/semantic_parser/`.\n",
    "\n",
    "`--nsp_data_dir`: Path to semantic parser datasets. Defaults to `~/minecraft/craftassist/agent/datasets/annotated_data/`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xQ8gahV8lkjW"
   },
   "source": [
    "You can now plug your own parsing models into the craftassist or locobot agents."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Craftassist_onboarding.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}